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Abstract 

This  paper  reports  ongoing  work  on  a  novel  mission- 
oriented  information  assurance  (I A)  assessment  ap¬ 
proach  that  contrasts  runtime  measurements  and  ob¬ 
servations  against  user-specified  requirements. 

1.  Introduction 

It  has  become  routine  to  have  multiple  security  me¬ 
chanisms  defending  advanced  mission-critical  systems. 
However,  quantifying  the  contribution  of  the  included 
mechanisms  (e.g.,  firewalls,  antivirus  scanners,  access 
control,  encryption  etc.)  or  the  underlying  survivability 
architecture  has  so  far  eluded  researchers  and  practi¬ 
tioners  alike.  Information  assurance ,  i.e.,  assurance 
that  the  security  mechanisms  are  effective,  and  the  sys¬ 
tem  can  be  entrusted  with  critical  information 
processing  tasks  is  largely  qualitative ,  and  is  estimated 
primarily  by  offline  analyses,  testing,  modeling  and 
experimentation.  Consequently,  during  mission  execu¬ 
tion,  arguably  the  time  when  it  is  most  critical  to  be 
assured  about  the  system,  IA  takes  on  an  all-or-nothing 
flavor  and  is  mostly  dependent  on  the  user’s  perception 
(i.e.,  either  the  user  continues  to  believe  the  offline 
assessment  or  not).  Runtime  variation  of  the  system’s 
assurance  due  to  attack-induced  failures,  environmental 
threats  (e.g.,  release  of  a  new  virus),  and  user-made 
changes  are  neither  well  understood  nor  considered. 

Without  a  runtime  assessment  of  the  system’s  as¬ 
surance  and  given  the  perception-based  and  all-or- 
nothing  view  of  IA,  war  fighters  and  system  owners 
might  incorrectly: 

•  Decide  not  to  use  the  system  fearing  compromise, 

•  Change  the  system  configuration  without  under¬ 
standing  the  impact  on  the  mission,  or 

•  Think  the  system  is  protected  (and  not  act  on  re¬ 
ported  events  or  continue  to  use)  when  it  is  not. 

All  of  these  cases  incur  risks.  With  the  increasing 
dependence  on  network  centric  information  systems, 
these  risks  cannot  be  left  unaddressed. 


In  this  paper  we  present  initial  results  of  our  ongo¬ 
ing  work  to  develop  a  continuous  assessment  frame¬ 
work  focused  on  the  assurance  of  mission  operations. 
In  this  context,  a  mission  refers  to  a  specific  set  of  tasks 
being  executed  by  an  information  system  to  support  a 
group  of  users  cooperating  to  achieve  a  common  objec¬ 
tive,  and  IA  refers  to  the  users’  level  of  confidence  that 
the  system  can  be  entrusted  with  their  respective  tasks. 
The  high  level  goal  of  this  research  is  to  demonstrate 
meaningful  and  continuous  mission-oriented  assess¬ 
ment  (CMA)  of  assurance.  More  specifically,  CMA 
aims  to  validate  the  following  claim:  information  sys¬ 
tems  can  be  instrumented  with  suitably  placed  probes 
and  aggregating  mechanisms  such  that  the  aggregating 
mechanisms  are  able  to  continuously  indicate  whether 
the  system  is  operating  at  a  required  level  of  assurance 
based  on  measurements  and  observations  reported  by 
the  probes.  An  additional  goal  is  to  support  I A  assess¬ 
ment-driven  adaptive  behavior  and  interoperation  with 
existing  QoS  mechanisms  [1]  enabling  QoS-IA  tra¬ 
deoffs  (e.g.,  sacrificing  encryption  for  faster  response). 

The  CMA  approach  is  a  significant  departure  from 
the  current  thinking  of  security  evaluation.  Initial  con¬ 
tributions  of  this  early  stage  research  include: 

•  A  taxonomy  and  organization  of  factors  that  con¬ 
tribute  to  mission-oriented  assessment  of  IA 

•  A  methodology  to  perform  the  assessment 

•  A  proof  of  concept  prototype  demonstrating  mis¬ 
sion-oriented  continuous  assessment  and  QoS  and 
I A  tradeoffs. 

2.  The  CMA  Approach 

CMA  first  captures  quantized  levels  of  IA  that  are 
acceptable  to  the  mission ;  then  identifies  observations 
and  measurements  that  can  be  collected  from  the  sys¬ 
tem  and  its  operating  environment;  and  uses  these  mea¬ 
surements  and  observations  to  determine,  on  a  conti¬ 
nuous  basis  throughout  the  mission,  whether  the  system 
is  operating  within  acceptable  levels.  Apart  from  mea¬ 
surements  that  are  readily  available  from  existing  En- 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
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terprise  System  Management  (ESM)  and  security  tools, 
CMA  interfaces  with  other  IA  and  resource  manage¬ 
ment  mechanisms  to  access  additional  information  that 
is  not  otherwise  visible. 

2.1  Multidimensional  Assurance  State  Space 

CMA  treats  acceptable  level  of  assurance  as  a  func¬ 
tion  of  regions  in  a  multi-dimensional  state  space. 

Time  is  a  key  dimension  in  this  space  because  as¬ 
surance  requirements  change  over  time  during  the  mis¬ 
sion.  Changes  in  assurance  requirements  can  be  based 
on  elapsed  time  (e.g.,  for  the  next  2  hrs  since  start)  or 
in  terms  of  mission  conditions  (e.g.,  from  start  until  an 
air  tasking  order  is  published). 

Each  mission  typically  has  multiple  stakeholders, 
and  each  stakeholder  typically  has  his  own  (time  vary¬ 
ing)  I A  requirements.  Therefore,  stakeholder  is  the 
second  dimension  of  the  multi-dimensional  space.  We 
are  initially  considering  three  representative  classes  of 
stakeholders:  commander  (with  an  ownership  stake), 
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Figure  1:  Stakeholders  and  sample  requirements 

warfighter  (end-user  stake)  and  operator  (system- 
administration  stakes).  Figure  1  shows  some  examples 
of  stakeholder  concerns. 

A  stakeholder  is  interested  in  specific  end-to-end 
capabilities,  subsystems  and/or  subsets  of  services  of¬ 
fered  by  the  system.  Therefore,  spatial  scope ,  capturing 
the  hosts,  networks  and  applications/services  that  are  of 
interest  to  a  stakeholder  is  the  third  dimension. 

Classically,  security  of  a  system  is  described  in 
terms  of  confidentiality  (C),  integrity  (I),  and  availabil¬ 
ity  (A)  [2].  Availability  and  confidentiality  are  defined 
in  terms  of  authorized  users,  but  do  not  consider  the 
strength  of  authentication  and  authorization  mechan¬ 
isms  involved  i.e.,  whether  authorization  was  based  on 
a  user-provided  password  or  validating  a  common 
access  card  (CAC)  (CAC  authentication  being  stronger 
than  password-based  authentication).  CMA  includes 
strength  of  authentication/ access  control  (A/A)  me¬ 
chanism  as  another  operationally  relevant  security 
attribute  independent  of  and  in  addition  to  C,  I  and  A. 
The  assurance  attributes  constitute  the  remaining  di¬ 
mension  of  our  multi-dimensional  assurance  space. 

We  use  projections  to  represent  and  analyze  ac¬ 
ceptable  levels  in  this  high-dimensional  assurance 
space.  For  example,  all  stakeholders’  interest  about  a 
specific  assurance  attribute  (e.g.,  C)  for  a  specific  spa¬ 
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tial  scope  (e.g.,  an  end-to-end  interaction  between  two 
applications)  over  time  can  be  captured  in  a  projection 
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Figure  2:  Projection  of  acceptable  levels 


like  the  one  shown  in  Figure  2.  To  cover  all  assurance 
attributes  for  the  given  spatial  scope  four  sets  of  such 
projections  will  be  needed.  In  Figure  2,  the  horizontal 
axis  represents  mission  progression.  Not  all  projections 
will  have  every  stakeholder  and  stakeholders  may  not 
have  requirements  for  every  point  on  the  time  axis. 

The  required  levels  of  assurance  at  a  given  point  in 
the  assurance  state  space  are  qualitative  and  ordered. 
This  is  deliberate,  because  while  quantifying  security  is 
hard  even  for  experts,  most  stakeholders  can  qualita¬ 
tively  express  their  time  varying  C,  I,  A,  and  A/A  re¬ 
quirements  for  the  system  components  and  services 
they  use  or  care  about.  The  following  nested  loop  de¬ 
scribes  the  assurance  requirements  capture  process: 

For  each  stakeholder 
For  each  element  in  his  spatial  scope 

For  each  assurance  dimension 

Capture  what  is  acceptable  over  time 
during  the  mission  in  relative  terms 

2.2  Metrics:  Measurements  and  Observations 

We  argue  that  no  fixed  list  of  metrics  is  universally 
applicable  and  assessment  must  work  with  the  security 
mechanisms  that  are  available  in  the  system  (i.e.,  as¬ 
sessment  cannot  prescribe  additional  mechanisms). 
Therefore,  the  best  that  can  be  done  is  to  define  the 
assessment  framework  in  terms  of  metric  classes,  and 
let  the  assurance  engineers  choose  for  each  class  the 
actual  properties  or  conditions  to  measure  and  observe 
from  what  is  available  in  the  system  at  hand.  Measura¬ 
ble  and  observable  quantities  are  generically  called 
“system  conditions”  (SC).  In  CMA,  the  SCs  are  orga¬ 
nized  in  a  shallow  hierarchy  (Figure  3)  with  two  broad 
categories  static  and  dynamic ,  and  leaves  (with  upper 
case  acronyms)  representing  the  CMA  metric  classes. 

2.2.1  Static  System  Conditions 

The  SCs  in  this  category  include  measurements  and 
observations  that  can  be  made  even  when  no  mission  is 
currently  running.  CMA  considers  two  classes  of  static 
SCs.  The  Process  and  Organizational  Maturity  (POM) 
class  focuses  on  the  maturity  of  the  software  and  secu¬ 
rity  engineering  process,  and  the  cultural  and  opera- 


tional  practices  of  the  organization.  The  other  class  in 
the  static  category  considers  Architecture  and  Infra¬ 
structure  Quality  (AIQ)-  how  the  system  is  constructed, 
especially  if  the  system  includes  mechanisms  to  pro¬ 
tect,  detect  or  manage  breaches  to  basic  security  prop¬ 
erties.  In  terms  of  why  these  classes  are  relevant  for 
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Figure  3:  System  condition  classes 


assurance  assessment,  consider  the  fact  that  a  large 
number  of  current  security  evaluation  methodologies 
(e.g.,  NSA  INFOSEC  Assurance  Capability  Model 
(2004),  CERT/CC  Security  Capability  Model  (2005), 
NIST  SP  800  (2001))  consider  process  and  organiza¬ 
tional  factors.  In  previous  work  [3]  we  showed  that  the 
resiliency  against  malicious  attacks  is  strongly  related 
to  the  system’s  survivability  architecture.  Some  securi¬ 
ty  checklists  do  include  architectural  quality  and  com¬ 
mon  Weakness  Enumeration  (CWE)  highlights  the  link 
between  software  quality  and  security  assurance. 


2.2.2  Dynamic  System  Conditions 

The  assurance  delivered  by  the  system  depends  on  a 
number  of  other  factors  that  can  only  be  measured  at 
runtime.  Most  of  the  runtime  metrics  affecting  assur¬ 
ance  originate  within  the  system  with  a  few  exceptions. 
Accordingly,  the  dynamic  system  conditions  are  further 
subdivided  into  internal  and  external  SCs. 

SCs  in  the  external  (EXT)  class  observe  relevant 
environmental  events  that  impact  the  system’s  security. 
Examples  include  CERT  or  vendor-issued  advisories, 
or  DEFCON  or  DHS  threat  level  type  indications  that 
may  imply  increased  risk  to  DoD  information  systems. 

CMA  defines  4  internal  SC  classes  (right  side  of 
Figure-3)  to  be  used  in  the  assessment  process.  A  large 
segment  of  such  information  about  the  system  and  its 
constituent  parts  (e.g.,  CPU  load,  memory  load,  net¬ 
work  load  etc.)  is  already  being  collected  and  measured 
routinely  today.  However,  in  typical  cases,  much  of  the 
information  gets  archived  or  fed  to  big  board  security 
incident  and  event  management  (SIEM)  stations  where 
there  exists  a  considerable  disconnect  between  the  sys¬ 
tem  operators  and  stakeholders’  requirements. 

First,  the  state  of  the  defense  mechanisms  (DEF 
ST  AT),  whether  they  are  functioning  at  their  intended 
configuration  is  a  primary  contributor  to  the  continuous 
assessment  process.  The  SCs  in  the  DEF  STAT  class 
observe  and  report  the  configuration  state  of  defense 


mechanisms  in  the  system.  Ideally,  changes  in  defense 
mechanism  configuration  resulting  from  operator  ac¬ 
tion  or  attack  activity  should  be  reflected  in  the  value 
reported  by  these  SCs.  The  key  challenge  to  realize  this 
ideal  is  to  make  the  observing  and  reporting  sufficient¬ 
ly  independent  of  the  defense  mechanism  such  that 
controlling  the  defense  mechanism  does  not  imply  con¬ 
trolling  the  monitor.  Enforcement  of  OS  and  network- 
level  isolation  policies,  stronger  process  authentication 
and  digital  signatures  can  be  employed  to  ensure  that 
the  adversary  cannot  easily  feed  incorrect  observation. 

Second,  the  state  of  system  resources  (RES  STAT) 
directly  impacts  the  availability  requirements,  and  since 
IA  mechanisms  in  general  need  resources,  the  ability  to 
meet  other  security  requirements  may  also  be  indirectly 
affected.  Therefore,  RES  STAT  SCs,  focusing  on  the 
status  of  computing  resources,  specifically  CPU,  mem¬ 
ory  and  the  network,  are  important  for  continuous  as¬ 
sessment.  Modern  hosts  and  network  equipment  (e.g., 
routers)  already  monitor  detailed  health  statistics.  Sys¬ 
tem  management  tools  (e.g.,  open  source  Nagios)  and 
protocols  (SNMP)  that  can  collect  and  distribute  them 
efficiently  are  widely  available.  SNMP  version  3  offers 
stronger  security  features  such  as  message  integrity, 
authentication  between  agent  and  server,  and  encryp¬ 
tion  that  the  RES  STAT  SCs  can  take  advantage  of. 

Third,  the  DEF  REP  class  includes  measurements 
and  observations  derived  from  defense  mechanism  re¬ 
ports.  The  DEF  REP  SCs  capture  information  about 
unexpected  or  suspicious  incidents  and  known  attack 
indicators  that  impact  the  mission.  Survivability  archi¬ 
tectures  or  security  management  tools  (e.g.,  open 
source  OS  SEC)  already  provide  a  way  to  interface  with 
host  based  security  mechanisms  in  a  secure  way  and 
present  processed  reports  via  a  server.  DEF  REP  SCs 
takes  advantage  of  existing  mechanisms  like  OSSEC. 

The  final  class  of  internal  system  condition  focuses 
on  effectiveness  of  defense  mechanisms  (DEF  EFF). 
Modern  systems  increasingly  combine  elements  of  pro¬ 
tection,  detection  and  adaptation  [4]  in  their  survivabil¬ 
ity  architecture.  Assessment  of  the  level  of  assurance 
must  consider  whether  defensive  responses  are  being 
effective.  Our  prior  work  [5]  on  monitoring  the  effec¬ 
tiveness  of  defensive  responses  provides  a  starting 
point  for  the  DEF  EFF  SCs. 

2.3  Assurance  Assessment  Scheme 

The  first  step  in  employing  CMA  in  a  given  system 
involves  a  thorough  system  analysis  to  find  out  what 
defense  mechanisms  are  in  place,  and  define  the  map¬ 
ping  from  the  assurance  levels  captured  in  the  projec¬ 
tions  described  in  Section  2.1  to  what  can  be  observed 
and  measured.  This  step  is  analogous  to  white  boarding 
[6],  and  will  produce  a  dependency  graph  (like  the  one 


in  Figure  4)  showing  the  spatial  scope  for  each  stake¬ 
holder  and  for  each  assurance  attribute  of  interest  along 
with  the  list  of  relevant  defense  mechanisms.  The  de¬ 
fense  mechanisms  can  be  protection-focused,  detec¬ 
tion-focused,  participate  in  defensive  responses  (adap¬ 
tation-focused)  or  can  be  any  combination  thereof.  The 
subsequent  steps  are  described  next. 

2.3.1  Identifying  What  to  Observe  and  Measure 

DEF  STAT,  DEF  REP  and  DEF  EFF  relate  to  the 
defense  mechanisms  in  the  graph  directly,  and  RES 
STAT  relates  to  the  spatial  scope  (i.e.,  host,  services 
and  networks).  In  CMA,  the  values  presented  by  the 
SCs  are  discrete  and  ordered,  which  means  raw  obser¬ 
vations  must  be  processed  before  reporting. 
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Figure  4:  Dependency  graph 

DEF  STAT  SCs  minimally  report  the  OFF  or  ON 
state  of  defense  mechanisms,  with  the  implicit  ordering 
that  ON  is  better  than  OFF.  Each  defense  mechanism 
(Ml ,  M2. . .)  present  in  the  dependency  graph  maps  to  a 
DEF  STAT  SC.  If  the  defense  mechanism  can  operate 
in  n  configurations  (e.g.,  different  key  lengths),  the 
corresponding  SC  projects  n  values,  one  corresponding 
to  each  configuration,  with  the  highest  value  assigned 
to  the  configuration  that  offers  the  strongest  security 
(e.g.,  128  bit  keys  are  stronger  than  64  bit  keys). 

The  value  presented  by  a  DEF  REP  SC  reflects  the 
severity  of  the  report  produced  by  the  corresponding 
defense  mechanism.  Such  reports  may  already  include 
a  severity  value.  Security  management  tools  also 
process,  correlate  and  rank  such  reports.  Where  possi¬ 
ble,  DEF  REP  SCs  interface  with  the  appropriate  me¬ 
chanisms  to  avoid  custom  ranking. 

Unlike  DEF  STAT  and  DEF  REP,  DEF  EFF  SCs 
are  not  tied  to  individual  defense  mechanisms.  In  the 
simplest  form,  a  single  DEF  EFF  SC  indicates  whether 
defensive  responses  mounted  by  the  system  are  being 
effective  or  not.  A  DEF  EFF  SC  has  an  internal  model 
of  “undesirable”  states  defined  in  terms  of  DEF  STAT 
and  RES  STAT  SCs.  When  an  undesirable  state  is 
reached,  it  starts  monitoring  DEF  REP  SCs  and  opera¬ 
tor  actions,  expecting  recovery  from  the  undesired 
state.  In  the  absence  of  recovery,  the  SC  begins  report¬ 
ing  lower  values  (i.e.,  defense  is  being  ineffective). 


Of  the  remaining  system  condition  classes,  POM  is 
similar  to  AIQ,  and  will  present  a  single  value.  If  the 
system  had  undergone  security  evaluation,  the  POM 
value  should  be  commensurate  with  the  results.  The 
AIQ  SC  comes  from  the  maturity  and  quality  of  the 
defense  mechanisms  and  the  number  of  defenses  effect¬ 
ing  individual  assurance  attributes  (for  a  given  spatial 
scope  for  a  given  stakeholder).  There  can  be  multiple 
SCs  in  the  EXT  class,  reporting  external  events  such  as 
publication  of  vulnerability  reports.  These  SCs  are 
identified  from  additional  annotations  (not  shown  in 
Figure  4)  of  the  assurance  dependency  graph  about  the 
host,  OS  and  application  services.  For  instance,  if  the 
system  predominantly  uses  Windows,  an  EXT  system 
condition  may  monitor  Windows  vulnerability  reports. 

Note  that  for  DEF  STAT,  DEF  EFF,  AIQ  and  POM 
(i.e.,  so  called  up  SCs)  higher  value  is  better;  whereas 
for  DEF  REP,  RES  STAT,  EXT  (i.e.,  so  called  down 
SCs)  higher  is  worse.  The  SCs  also  do  not  need  to  have 
the  same  range  of  values  (0-1,  0-5,  1-10  etc.).  Some 
measurements  are  continuous  at  the  lowest  level  (e.g. 
CPU  load),  but  are  quantized  based  on  thresholds  or 
reported  as  a  derived  discrete  measure  (e.g.,  trend  of 
average  load  crossing  a  threshold). 

In  a  given  system,  not  all  SC  classes  may  be  needed 
or  supported.  For  instance,  the  deployment  environ¬ 
ment  may  not  permit  live  subscription  to  vulnerability 
reports  (i.e.,  no  EXT  SCs).  The  assurance  dependency 
graph  may  not  be  full  or  exhaustive  either.  For  exam¬ 
ple,  there  may  not  be  any  defense  mechanism  covering 
a  specific  service  whose  security  is  of  interest  to  one  or 
more  stakeholders-  a  system  deficiency  identified  as  a 
byproduct  of  CMA  requirements  analysis.  CMA  is 
about  assessing  individual  systems,  as  long  as  the  set  of 
SCs  is  used  in  an  internally  consistent  way  within  the 
entire  assurance  space  for  the  mission,  such  deficien¬ 
cies  are  immaterial. 

Once  the  SCs  are  identified,  the  system  needs  to  be 
instrumented  so  that  measurements  and  observations 
are  projected  via  system  condition  objects. 

2.3.2  Mapping:  Required  vs.  Supportable  Levels 

Once  the  SCs  are  identified,  the  next  step  is  to  per¬ 
form  a  validity  check:  can  the  system  support  the  de¬ 
sired  number  of  levels?  A  stakeholder  may  expect  3 
levels  of  confidentiality  in  a  link,  but  the  system  can  be 
configured  to  offer  only  two  (say  encryption  OFF  or 
ON).  All  projections  need  to  be  checked  and  identified 
issues  need  to  be  reconciled.  Because  multiple  layers  of 
defense  are  expected  in  a  survivable  system,  it  is  likely 
that  the  system  can  be  configured  to  offer  more  levels 
than  what  the  stakeholder  required.  This  provides  the 
opportunity  to  group  multiple  configurations  in  a  level. 
As  an  example,  consider  a  stakeholder’s  interest  in  the 


confidentiality  of  an  end-to-end  capability  that  uses  a 
link  connecting  his  application  with  a  remote  service. 
The  stakeholder’s  confidentiality  requirement  is  ex¬ 
pressed  in  terms  of  3  levels  (i.e.,  High,  Medium,  Low), 
and  the  physical  link  can  either  be  clear  text  or  en¬ 
crypted  and  the  application  interaction  can  also  be  en¬ 
crypted  or  clear  text  (e.g.,  http  or  https).  Collectively, 
there  are  4  different  ways  the  stakeholder  can  operate 
(link  unencrypted,  http),  (link  encrypted,  http),  (link 
unencrypted,  https),  (link  encrypted,  https).  In  this  case 
we  have  the  option  to  select  which  configurations  cor¬ 
respond  to  each  of  the  3  stakeholder  levels.  Since  the 
SCs  present  discrete  and  ordered  values,  the  above  4 
configurations  can  be  described  using  two  binary  va¬ 
lued  DEF  STAT  SCs,  link  enc  and  app_enc,  as  (0,0), 
(1,0),  (0,1)  and  (1,1)  respectively.  Because  individual 
system  condition  values  are  ordered,  these  configura¬ 
tions  are  only  partially  ordered  with  (1,1)  being  the 
highest  and  (0,0)  being  the  lowest,  with  the  remaining 
2  (i.e.,  (1,0)  and  (0,1))  in  the  middle  with  no  ordering 
among  themselves.  One  possible  mapping  of  the  3  re¬ 
quired  levels  can  be  {low  =  (0,0),  medium  =  (1,0)  or 
(0,1),  high  =  (1,1)}.  Alternatively,  mission  require¬ 
ments  may  declare  (0,0)  unacceptable,  which  would 
lead  to  a  grouping  like  {low=(0,l),  medium=(l,0), 
high=  (0,1)}. 

2.3.3  General  Structure  of  the  Assessment  Function 

In  CMA,  the  assessment  function  is  the  logic  that 
maps  the  observed  SC  values  to  the  assurance  levels 
specified  by  the  stakeholders.  In  general,  the  assessed 
level  of  assurance  for  stakeholder  H,  for  entity  (spatial 
scope)  E  of  interest  to  H,  and  for  the  security  attribute 
A  at  any  point  in  the  mission  is  a  function  of  a  baseline 
value  and  a  variance  that  modifies  the  baseline: 

Assessed  hevelhEA  -  LHEA(Baselir\9f  Variance) 
Baseline  refers  to  the  (assessment  of  the)  idealized  lev¬ 
el  of  assurance  that  the  system  is  designed  to  offer  in  a 
given  configuration.  Variance  captures  the  impact  of 
internal  changes  caused  by  attacks  or  user  actions  as 
well  as  external  events.  Baseline  is  a  function  fCO  of 
DEF  STAT  SCs,  and  optionally  AIQ,  POM,  and  Va¬ 
riance  is  a  function  gCO  of  RES  STAT  system  condi¬ 
tion,  and  optionally  DEF  REP,  DEF  EFF,  EXT,  AIQ, 
POM.  In  other  words: 

Baseline  =  f  (DEF  STAT,  AIQ,  FOM) 

Varia-ncg  =  'JIES  STAT ,  DEF  REP,  DEF  EFF,  EXT,  AIQ,  POM) 

The  functions  fCO  and  g(0  are  context  dependent 
to  account  for  mission  customizability.  DEF  STAT  and 
RES  STAT  are  the  only  mandatory  inputs,  but  other 
inputs  are  also  accommodated  when  available.  We 
have  already  demonstrated  meaningful  results  using 
only  the  mandatory  inputs. 


Evaluating  f(0  using  DEF  STAT  amounts  to  per¬ 
forming  a  membership  operation  between  the  currently 
observed  values  of  DEF  STAT  SC  (relevant  to  H,  E 
and  A)  to  the  levels  (as  explained  in  Section  2.3.2)  in 
the  state  space  of  this  set  of  SCs.  The  variance  can  be 
positive  or  negative  -  it  can  either  add  to  or  diminish 
from  the  baseline  assessment  depending  on  the  situa¬ 
tion.  Publication  of  a  new  Windows  exploit  (captured 
by  an  EXT  system  condition)  can  cause  the  assessment 
of  all  attributes  for  spatial  scopes  that  include  Windows 
hosts  to  be  lower  than  the  baseline.  On  the  other  hand, 
closing  down  a  vulnerable  port  on  Hostj  (captured  by  a 
DEF  REP  system  condition)  can  cause  the  assessment 
of  all  attributes  for  the  spatial  scopes  that  include  Hostj 
to  rise.  The  function  DHEiA(-f-)  that  combines  baseline 
with  variance  is  also  context  sensitive.  For  example,  a 
variance  based  on  RES  STAT  usually  has  a  multiplica¬ 
tive  effect  on  assessment  of  availability  only,  whereas  a 
variance  based  on  EXT  e.g.,  vulnerability  report  VR  can 
have  a  negative  additive  effect  on  the  assessment  of 
security  attributes  noted  in  VR. 

The  fact  that  there  is  no  universal  model  for  compu¬ 
ting  fCO,  gCO  and  DHfEfA(-r)  is  a  continuing  challenge. 
To  address  this  challenge,  CMA  accommodates  user 
defined  context  models.  As  long  as  the  model  is  consis¬ 
tently  used  in  the  system,  CMA  provides  a  way  to  as¬ 
sess  the  runtime  assurance  state. 

3.  Current  Prototype 

We  have  an  evolving  proof  of  concept  assessment 
framework  that  we  use  for  demonstrating  and  evaluat¬ 
ing  CMA.  The  aggregator  nodes  performing  assess¬ 
ment  for  different  stakeholders  are  called  the  black¬ 
boards  that  communicate  with  each  other  using  web 
services.  Assessment  rules  for  fCO  etc  are  defined  in 
TU  Prolog,  invoked  from  Java.  The  first  version  of  the 
prototype  focused  on  the  assessment  function  and  dem¬ 
onstrated  continuous  assessment  for  a  single  stakehold¬ 
er  using  only  the  mandatory  DEF  STAT  and  RES 
STAT  SCs  in  a  simulated  mission  context.  Mission 
progress  was  simulated  by  advancing  time  and  by  gene¬ 
rating  mission  events.  Measurements  and  observations 
were  changed  by  injecting  values  from  the  simulation 
control  console.  The  first  prototype  verified  the  feasi¬ 
bility  of  a)  the  structure  and  methodology  to  capture  IA 
requirements,  b)  the  DHEA(~y)  assessment  scheme,  and 
c)  web  services-based  aggregator  interactions.  The 
second  version  of  the  prototype  deployed  blackboards 
over  a  distributed  set  of  hosts  and  demonstrated  a)  in¬ 
tegration  with  COTS  system  and  security  management 
(e.g.,  OSSEC  and  Nagios)  to  obtain  metrics,  and  b) 
peering  of  aggregated  information  for  assessment. 


4.  Related  Work 

Many  manual  IA  assessment  techniques  today  rely 
on  vulnerability  databases  (e.g.,  CVE  [7])  or  security 
policies  and  guidelines  (e.g.,  EAL  levels  of  common 
criteria  [16]).  Vulnerability  reports  may  come  from 
vendors  and  organizations  like  CERT.  There  are  repo¬ 
sitories  like  Common  Vulnerabilities  and  Exposures 
(CVE)  [7]  that  organize  and  cross-reference  reports 
from  disparate  sources.  Common  Vulnerability  Scoring 
System  (CVSS)  is  an  example  of  an  extensible  standard 
for  vulnerability-based  scoring  using  temporal  or  envi¬ 
ronmental  factors  [8].  On  the  automated  assessment 
side,  Security  Content  Automation  Protocol  (SCAP) 

[9]  defines  six  constituent  standards,  including  CVE 
and  CVSS,  for  scoring  systems  based  on  compliance  to 
specified  configuration  and  the  vulnerability  impact.  In 
principle,  vulnerability-based  assessment  can  be  done 
continuously  when  the  system  is  in  operation  and  cus¬ 
tomized  for  a  mission,  but  it  is  typically  done  by  an 
analyst,  scoring  a  specific  part  or  subsystem  at  a  time. 
In  CMA,  vulnerability  is  one  of  the  many  factors  con¬ 
sidered  for  assessing  assurance. 

Several  NIST  publications  (e.g.,  NIST  Special  Pub. 
800-55 [10],  Federal  Information  Processing  Standard 
(FIPS)  140  [11]),  the  orange  book  and  its  successor 
common  criteria  [12]  offer  examples  of  models  and 
rules  that  are  used  to  assess  the  security  level  of  infor¬ 
mation  systems  or  system  components.  Common 
Weakness  Enumeration  (CWE)  came  out  of  the  need  to 
formally  categorize  security  weaknesses  [13],  and  can 
also  be  used  in  a  check-list  oriented  assessment.  But 
the  assessment  process  needs  human  experts,  is  not 
runtime  and  does  not  have  a  mission  focus. 

Dependability  centric  measurements  that  require 
long  term  observation  of  the  system  or  model-based 
studies  to  compute  metrics  like  mean  time  to  attack 
(MTTA),  mean  time  to  failure  (MTTF),  or  mean  time 
to  recovery  (MTTR)  offer  another  way  to  rate  a  system. 
However,  such  measures  are  not  suitable  for  systems 
that  are  safety  critical,  systems  that  have  not  been  dep¬ 
loyed  in  a  comparable  environment,  and  more  impor¬ 
tantly,  for  making  dynamic  and  run  time  adaptation 
decisions  (one  of  our  major  goals). 

Red  teaming  [14]  is  an  approach  that  is  often  used 
to  assess  the  quality  of  defense.  While  useful  to  identify 
system  flaws,  rating  a  system  based  on  red  team  evalua¬ 
tion  can  be  misleading  because  the  capability,  motiva¬ 
tion  and  resources  of  the  red  team  vary. 

Quality  of  protection  (QoP)  uses  protection  as  a 
quantitative  measurement  across  an  entire  system.  QoP 
provides  an  umbrella  for  a  number  of  different  security 


related  properties  (e.g.,  mapping  multi-level  security  to 
QoP  [15],  vulnerabilities  and  best-practices  to  model 
QoP  [16]).  However,  the  ratings  are  still  static. 

5.  Conclusion  and  Next  Steps 

CMA  makes  it  possible  for  mission  stakeholders  to 
get  a  continuous  status  of  how  the  system’s  IA  mechan¬ 
isms  are  holding  up  against  their  requirements  and  pro¬ 
vides  a  risk  assessment  implication  of  their  ac¬ 
tions/choices.  Even  the  absence  of  assessed  value  (due 
to  disruption  in  the  flow  of  measurement  or  observa¬ 
tion)  is  valuable  to  the  stakeholders.  CMA  also  pro¬ 
vides  the  foundation  of  “managed  quality  of  IA”  where 
IA  and  service  delivery  can  be  actively  managed  at 
runtime,  trading  off  IA  and  QoS  as  necessary. 

Although  CMA  work  is  in  its  early  stages,  initial  re¬ 
sults  are  promising.  Informed  by  early  results,  we  are 
enhancing  the  CMA  prototype  to  support  additional 
stakeholders,  more  complex  missions  and  demonstrate 
assessments  involving  additional  metric  classes.  We  are 
also  continuously  testing  and  evaluating  our  prototype. 
A  red  team  type  evaluation  to  test  the  CMA  approach 
and  meaningful  assurance-service  delivery  tradeoff 
under  adverse  conditions  is  planned. 
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